Contextual Media Retrieval Using Natural Language Queries

The 21st century has seen a rapid increase in the abundance of mobile devices with cameras. This, along with the evolution of digital photography and the internet, has presented mankind with a virtual mine of media content. The increasing number of images and videos rich with metadata (timestamps, GPS location, camera orientation etc.) has the potential to act as a collective memory dispersed in space and time. Put to good use, these can lead to an application which can be deemed as the closest approximation to spatio-temporal exploration of the world. However, such a virtual exploration is practically impossible without human interaction with the computer. In such a scenario, research on Ubiquitous Computing, supported by wearable technology, tries to make these interactions as friction-less as possible. In our work we develop a query-retrieval system with which users can browse through the collective memory with natural language voice queries and enjoy the visual treat presented as egocentric images and videos. We have extended a state-of-the-art semantic parser to suit the dynamic egocentric environment that a spatio-temporal exploration calls for. Since the Google Glass has made humancomputer interactions very easy (by allowing hands-free interactions through voice commands), we demonstrate our query-retrieval system on the Google Glass. The novelty of our work lies in the development of a visual collective memory, in the adaptation of a parser to handle spatial and temporal references in English language questions, in the addition of context or egocentrism to media retrieval and in the demonstration of a query-retrieval system in a dynamic environment.